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Introduction 



When to Use This 
Handbook 



Project Evaluation 
is Important to the 
Library of Michigan 
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This handbook has been developed by the Library of Michigan with the 
encouragement of the U.S. Department of Education, It will guide you in 
planning and conducting evaluations of your grant programs. The hand- 
book is intended to be practical, easy to use, and relevant to the evalua^ 
tion needs of library staff. Following comments on the use of the handbook 
and the stakeholder approach to evaluation, a step-by-step method is used 
to present the information. Seven steps for project evaluation take the 
reader from start to finish. These steps are explained in detail later in this 
publication. 



It is important to understand the context in which this handbook is 
meant to be used. Evaluation is only one component of project planning 
and the Library Services and Construction Act (LSCA) application pro- 
cess. This handbook covers only the development of your project evalua- 
tion, it is not meant to be an overall guide to preparing an LSCA grant 
application. Prior to applying the seven-step method presented here, you 
should do all of the following: 

1. Recognize unmet community needs which library services and/or 
programs may meet; 

2. Read all documents in the current LSCA application package; 

3. Match the needs you have identified with LSCA funding areas for 
potential project development; and 




4. Attend a Library of Michigan grant writing workshop to improve your 
overall understanding of the application forms and guidelines. 



In order to receive the most benefit from this handbook, we suggest 
that you have a project in mind, complete with documentation of the 
needs it will address, as well as the project goal and objectives. At this 
point you are ready to apply the stakeholder approach as you develop your 
project's plan for evaluation. 



The Library of Michigan is responsible for the administration of Library 
Services and Construction Act Titles I, II, and III. These grant programs 
award federal funds to support a variety of library services. The number of 
grants awarded has grown considerably over the past five years as evi- 
denced by the 147 projects funded in 1993. In response to this growth, the 
Library of Michigan has offered grant writing workshops, enhanced the 
clarity of the application forms, and added greater detail to the grant 
guidelines. These efforts were undertaken to make applying for federal 
grants more manageable for applicants and the Library of Michigan alike. 

State Librarian James W. Fry recently said, “Evaluation is the watch- 
word for the 90's when using public funds." This applies to the potential 
grant recipient as well as to the Library of Michigan. The Library of Mich- 
igan seeks well-planned project proposals supported by meaningful evalua- 
tions. Evaluation of grant-funded projects is important to the Library of 
Michigan for many reasons. The results of project evaluation are used for: 



.7 



1 



♦ Identifying mcxiel projects to serve as examples for other libraries 

♦ Completing reporting requirements of the U.S. Department 
of Education 

♦ Ensuring development of a well thought out project by the applicant 

♦ Documenting to Congress the continued need for LSCA funding 

Upon becoming a subgrant recipient, you are responsible for the wise 
expenditure of federal dollars. Likewise, the Library of Michigan is ac' 
countable to the federal government for the effective use of Michigan’s aL 
lotment of LSCA funds. We share responsibility for the thoughtful expen^ 
diture of federal funds with you and because of that, the Library of 
Michigan is fully committed to strong, purposeful evaluations of every 
LSCA'funded project in Michigan. 



Project Evaluation 
is Important to 
Your Library 



In addition to meeting the requirements of the Library of Michigan, 
you should evaluate programs to plan effectively, to improve programs, 
and to increase program impact. A program or project is a set of activities, 
services, and materials organized to achieve a specific goal, supported 
by project objectives. Examples of LSCA-fiinded projects include an 
event sponsored by a public library or an outreach service of the library or 
of the public library cooperative. 

One purpose of this handbook is to assist library staff in learning the 
simple steps that can provide a meaningful evaluation. Some library staff 
have been reluctant to evaluate programs. Common reasons for this reluc- 
tance include seeing no direct benefits for the library, believing there is a 
lack of knowledge or time to conduct a credible evaluation, or finally, fear- 
ing that evaluation results could show failure rather than success. 

Evaluation of a project or a program can produce information that is 
useful to all interested parties. You may want to adjust a program in mid- 
stream to make it more effective, or decide whether to repeat a special 
event a second year, or approach a funding source to expand a given ser- 
vice. In any of these cases, information on the impact of your project — 
evaluation data — will be of great help. Effective evaluation requires addi- 
tional investments of time and resources; however, the benefits of 
evaluation are worth these extra efforts. 



The Stakeholder 
Approach to Project 
Evaluation 



The Library of Michigan encourages an evaluation process that involves 
a representative group of those people who manage, receive services, or 
otherwise have an interest in a project. This is a stakeholder approach — 
involvement of those people who have a stake in the evaluation findings 
and the success of the project. Library staff, library users and community 
leaders are all potential stakeholders. The identification and use of stake- 
holders is the cornerstone of the evaluation technique presented in this 
handbook The Library of Michigan rating scale used to rank LSCA grant 
applications is weighted to award up to 25 of the 100 possible points in 
recognition of a strong evaluation component. 
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The involvement of stakeholders in your project’s evaluation will have 
many benefits. Overall, the varied points of view and expertise these indi- 
viduals will bring to the project are likely to be very helpful. While the 
grant administrator has responsibility and authority for all aspects of the 
project, stakeholders can be an excellent resource whose advice many prove 
invaluable in planning evaluation. 

Other benefits of the stakeholder approach include the added legitimacy 
that involving a broader group of individuals brings. This same group, ex- 
tending beyond the library and into the community, may also bring added 
community support before, during and following the project’s life. 

The Library of Michigan strives to make the grant application process 
understandable to applicants. We also want to help you to plan strong 
projects and to write effective proposals. The better organized the plans are 
for a project, the more likely it is that the project v/ill be a success. Project 
evaluation is an integral part of the Library of Michigan LSCA application 
form. Evaluation should also be an essential part of all project planning. In 
reviewing this handbook, notice that the planning process for a project and 
planning for the project’s evaluation should occur simultaneously. Conduct- 
ing evaluation as a continuous process during the life of a project will help 
to insure the success of your project. 

A final word about evaluation before you begin to read this handbook — 
there is no reason to restrict evaluation to LSCA grant projects. While this 
handbook has been developed within the context of the Library of 
Michigan’s LSCA grant program, the stakeholder approach to evaluation 
can be applied to any program or project your library is undertaking. The 
desire to offer useful and successful services to their users is common to all 
librarians. Evaluation can serve as the key ingredient in your recipe for suc- 
cess by providing information for project fine tuning as well as the docu- 
mentation needed to seek additional funds. Feel free to use the ideas pre- 
sented here in many areas of your library’s operation. 



SEVEN STEPS of 
Project Evaluation 




The seven steps for project evaluation that take the reader from start to 
finish are: 

1. Develop a Preliminary Project Plan 

2. Identify the Project Stakeholders 

3. Hold a Meeting of Project Stakeholders 

4. Redefine the Project Proposal 

5. Collect and Analyze Evaluation Data 

6. Summarize Evaluation Data for Stakeholders 

7. Prepare Final Evaluation Report 
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Step One: Develop a 
Preliminary Project Plan 



Step Two: Identify the 
Project Stakeholders 



Step Three: Hold a 
Meeting of Project 
Stakeholders 



A* Prepare an agenda 
for the meeting 
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As indicated earlier in this handbook, your first step must be to exam- 
ine your program and service needs in comparison with current LSCA 
funding areas. After identifying the areas eligible for LSCA funding, begin 
thinking about each individual project’s goal and objectives. Long before 
you begin to complete an application form, think through your project 
carefully, beginning to identify what you will do to achieve each objective 
and who will be involved in the project. 

The cornerstone of project evaluation design *s the participation of the 
stakeholders; those people who have a stake in the evaluation findings. Li" 
brary staff, library users, library board members, and other community lead- 
ers are all potential stakeholders in a program or project. These individuals 
may be involved in the project’s implementation, may be in decision-mak- 
ing positions for future project funding, or may be potential recipients of 
project services. 

Evaluations should be designed to be both useful and used. Before you 
begin, ask yourself these questions: 

• How can I design the evaluation? 

• How should I collect and analyze the data? 

• How should the findings be reported; so that my stakeholders can 
plan, implement, and increase the impact of our program? 

Answering thse questions requires addressing the needs of stakeholders 
and being responsive to the ways iii which information is used within your 
organization. 

Early and continuous involvement by representatives of the various 
stakeholder groups will increase the likelihood that your evaluation find- 
ings will also be used by the stakeholders. Their participation means that 
they will have a say in the information to be collected; they will feel a 
sense of ownership in both the evaluation and in the program or project 
itself. 

A successful meeting with stakeholders will require that these three sub- 
steps are completed. You will need to: 

A. Prepare an agenda for the meeting 

B. Review the Project Evaluation Worksheet prior to the meeting 

C. Hold the meeting, following the agenda items 

Invite representatives of each of your stakeholder constituencies to a 
meeting. At the time these invitations are issued, you may want to seek 
input from these individuals regarding additional stakeholder constituen- 
cies for your consideration. At this meeting, plan to discuss the overall 
purpose of the proposed project or program, the tentative outline of 
project activities, and the purpose of the evaluation. A tentative agenda, 
ready to be individualized with your library name, meeting place and time 
for this first meeting, is included in Appendix A. It is helpful to mail cop- 
ies of the agenda to participants in advance of the meeting. 
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B* Review the Project 
Evaluation Worksheet 
prior to the meeting 



A Project Evaluation Worksheet (Appendix B contains a completed 
sample worksheet) is designed to help you ask and answer all the questions 
necessary to create a project evaluation at this first meeting. To ready 
yourself for the meeting, you should develop the basic details of the pro- 
posed project, be prepared to share this information with stakeholders, and 
have carefully reviewed the Project Evaluation Worksheet. As the grant 
administrator, you should lead your stakeholders through a review of the 
worksheet, and have specific ideas on the desired outcomes of these efforts. 
Resist the temptation to skip ahead in the questions. The sequence of 
these questions is important. Take time when you start designing your 
evaluation to answer each one as fully as possible. The completed Project 
Evaluation Worksheet will become a component of your LSCA grant ap- 
plication. The application package sent to all public libraries now con- 
tains the Project Evaluation Worksheet as one of the required items to be 
completed along with the abstract, budget, project narrative and other pro- 
posal components. 



C. Hold the Meeting, 
Following the 
Agenda Items 



Library Project 
Planning Meeting 

Tentative Agenda 

. 1. . latKodni^ Mtci^iig 

2. ' ■■ Review, ' , 

■ ' A i: niicw . ItiVRfit* 

« Siimmkitiit 'Bffmi t6 Ptt^ 

3. Deccribc the Proposed Project 
« Goal and Objectives 

* Activities 

* Budget 

* Evaluation 

4. Discuss and Complete Project Evaluation 
Worksheet 

5. Determine On-Going Role for Meeting 
Participants 

6. Schedule Next Meeting Date 

7. Adjournment 



Agenda Item 1 . Introduce 
Meeting Participants 



Agenda Item 2. Review 
Purpose of Meeting 
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Introduce each of the participants, including the agency or constituency 
which they represent, and their relationship to the proposed project. 



The better prepared you are to present the project plan and the overall 
purpose of the meeting, the more productive the group’s efforts will be. Be 
clear about what the project or program will try to achieve, as well as the 
reason you are meeting with the group. This agenda item allows you to 
cover five important points while describing the purpose of the meeting: 
identifying the source of the grant funds, discussing community need' in 
relationship to grant funding categories and your library’s mission, present- 
ing the broad project concept, describing the stakeholder evaluation pro- 
cess, and summarizing your efforts to date* 
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Agenda Item 3. Describe the 
Proposed Project 



In order to help your group to get a firm understanding of the proposed 
project, outline the goal and project objectives, the activities you expect 
will take place to achieve each objective, and the budget. You may want 
to re-empha.ize the importance of the group’s work by ending with the po- 
tential for project evaluation. The old adage, “If you don’t know where 
you’re going, how will you know when you get there?” applies here. Un- 
derstanding your overall project goal and knowing the objectives and their 
desired outcomes is essential for both you and your group of stakeholders. 



Agenda Item 4. Discuss and 
Complete Project Evaluation 
Worksheet 



Library Project 
Planning Meeting 

Tentative Agenda 

1. Introduce Meeting Participants 

2. Review Purpose of Meet. 

• Identify Source of Gri t Funds 

• Discuss Community Need in Relationship 
to Grant Funding Categories 

• Present Overview of Project Concept 

* Describe Stakeholder Evaluation 
Process 

* Summarize Efforts to Date 

3. Describe tbe Proposed Project 
; « Go#i Directives 

♦ Aaivititt 

' /> BviittMtioik 

'4C and Complete Project Evaltittion 

Woeksh 

5. Determine On-Going Role for Meeting 
Participants 

6. Schedule Next Meeting Date 

7. Adjournment 
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The Project Evaluation Worksheet which will be completed during this 
meeting is a key planning document for project administration. It helps to 
set the foundation blocks for what the project will be and how its success 
will be assessed. The wwksheet (See Appendix B for a completed sample) 
identifies five questions which must be answered in developing your 
project evaluation: 

I. What questions will the evaluation answer? 

II. What sources of data will be used to answer the questions? 

III. What methods will be used to collect the data? 

IV. In what forms and to whom will the evaluation be reported? 

V. How will the evaluation results be used? 

Determine what questions will be used to guide the measure of project 
success before identifying what data you will collect. Avoid the temptation 
of determining the data that can be easily gathered for the project and de- 
signing evaluation questions to match. The key step is identifying the 
questions that the evaluation should answer in order to determine the im^ 
pact of the project. Your project evaluation should ultimately seek to answer the 
question f “So what?; what difference did this project make?'' This is a role for 
the stakeholders. Ask them what they w^ant to know and why this infor- 
mation is important. 

For example, evaluation of a project to develop a special library col- 
lection for older adults might address the following questions: 

♦ What were the characteristics of the people who 
used the collection? 

♦ Was the collection used by older adults in the tar- 
get community? 

♦ In what ways did the new collection have an im- 
pact on the library? 

♦ In what ways did the collection have an impact 
on the local communities? 

♦ What do older adults think of the library and the 
new collection? 

♦ Is the cost of the collection worth the benefits 
that it provides? 

Both objectives and the methodology or activities in your project pro- 
posal may be relatively easy to measure. For example, evaluation data can 
be gathered to give you counts of items purchased, numbers of individuals 
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Agenda Item 4. Discuss and 
Complete Project Evaluation 
Worksheet 

(Continued) 



in attendance at a program, and so forth. When looking more broadly at 
your project or program goal, you and the stakeholder group must identify 
questions whose answers will document real success or project impact, but 
may be more difficult to measure. The questions you seek to answer should 
go well beyond counts of materials purchased or numbers of individuals in 
attendance at an event. 

For example, if the goal of your project is to enhance the image 
of the library as an information provider in the community, po- 
tential indicators of achievement of this goal include: 



Library Project 
Planning Meeting 

Tentative Agenda 

1. Introduce Meeting P»rticipant» 

2. Review Purpote of Meeting 

• Identify Source of Grant Fundt 

• DiscuKt Community Need in Relationahip 
to Grant Funding Categor'^rs 

• Preaent Overview of Project Concept 

• Describe Stakeholder Evaluation 
Process 

• Summarize Efforts to Date 

3. Describe the Proposed Project 

• Goal and Objectives 

• Activities 

• Budget 

• Evaluation 

<4*, DUciitift Complete Project Eveiuttion 
Worksheet 

5. Determine On-Going Role for Meeting 
Participants 

6. Schedule Next Meeting Date 

7. Adjournment 



♦ Has library funding improved? 

♦ Have general community attitudes toward the 
public library changed? 

♦ What is the reputation of the library among 
community leaders? 

♦ What are the library staff attitudes toward the 
library's services? 

♦ In what ways is the library used by the local 
community? 

Let the evaluation questions determine the best methods and proce- 
dures to use in data collection. Be sure to identify your project's evaluation 
questions before selecting any collection methods or data sources. Avoid 
reversing this order by attempting to determine how you will collect the 
data before developing the questions. If you are asking, **How many people 
in the target population were affected by this program?" then records of 
program usage and demographics of participants should be reviewed. If you 
are asking, "How does this program affect library users?" then a survey of 
all users or interviews of small groups of users are possible strategies. 

People, library statistics, reports, and program records are all potential 
data sources to answer your evaluation questions. Frequently many sources 
will help you answer a question. By identifying the best data sources for 
answering each question, you will also be able to select the most effective 
data collection methods. 

For example, to answer the question, "How has our new library 
facility improved the quality and quantity of service to library' us- 
ers?," data could be collected from the following sources: 



♦ Circulation records 

♦ Comments by library users 

♦ Applications for library cards 

♦ Community groups that use library facilities 

♦ Head count of library users 

♦ Comments by public officials 

♦ Comments by outside consultants 

♦ Comments by library staff 
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Agenda Item 4 ■ Discuss and 
Complete Project Evaluation 
Worksheet 

(Continued) 
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Library Project 
Planning Meeting 

Tentative Agenda 

I* Introduci; Meeting Participant* 

2. Review Purpose of Meeting 

* Identify Source of Grant Funds 

* Discuss Community Need in Relationship 
to Grant Funding Categories 

* Present Overview of Project Concept 

* Describe Stakeholder Evaluation 
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* Summarize Efforts to Date 
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* Activities 
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Agenda Item 5, Determine 
On-Qoing Role for 
Meeting Participants 



Agenda Item 6, Schedule 
Next Meeting Date 



Step Four: Redefine the 
Project Proposal 
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While the stakeholder group may identify multiple sources, you should 
select a limited number of reliable sources for use in answering any single 
question. Choices should be made based on the sources which will provide 
you with the best data. In deciding, you will also want to take into consid- 
eration available staff time, reliability of data, financial resources, timeli- 
ness of information, and deadlines for reporting the findings. 

Useful evaluations can be conducted using any one of a wide variety of 
data collection methods. The Data Collection Methods table in Appendix 
C lists seven general categories of methods that can be used to evaluate li- 
brary programs or projects. Your choice depends on the question that you 
are asking and the sources from which you want to collect the data. You 
can use the table to determine which methods are best to use in your situ- 
ation. Many excellent resources on evaluation methods are available. A 
bibliography that includes a few of these sources is provided at the end of 
this handbook in Appendix D. 

The last two questions to be answered on the Project Evaluation 
Worksheet relate to matters following the data collection phase of your 
work. As the stakeholders discuss their interests and responsibilities in the 
proposed project, help them to identify the most useful formats in which 
evaluation information should be reported, how the information will be 
used, and the potential outcomes of their deliberations after project evalu- 
ation. Appendix B provides a sample of possible formats for reporting data 
and ideas on the possible use of evaluation results. Stakeholders may not 
be aware of the potential for project fine tuning, extension, and/or expan- 
sion that this information may provide. As you complete the Project 
Evaluation Worksheet, remember that it will become an integral compo- 
nent of your LSCA grant application. 

Your final agenda item, other than setting another meeting date, should 
be identifying the ongoing role, if any, for the stakeholder group. Appen- 
dix E lists many other roles for members of this group. At the very least, 
invite a smaller core group to serve as an ongoing advisory committee for 
the project. This group can assist in pilot testing evaluation methods, con- 
ducting some phases of actual project evaluation, and providing advire on 
mid-project fine tuning. 

You may actually be setting dates for two meetings. While your smaller 
core group may need to meet again in the near future to discuss the devel- 
opment of evaluation instruments or related matters, the entire stakeholder 
group should plan a meeting at the conclusion of the project at which 
time you share evaluation results and the draft of final project reports. 

As we have already said, your project plan is likely to benefit from the 
varied experiences and points of view of the stakeholder group. It is prob- 
able that following the completion of the stakeholder meeting, new ideas 
will have surfaced, inspired by the proposed project. As grant administra- 
tor, you have the ultimate responsibility of evaluating all recommendations 
and advice, and consolidating it into a well-written project proposal which 
has the strongest likelihood of meeting the project goal. 
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BEST COPY AVAILABLE 



After completing the review and revision of all aspects of the proposal, 
you will seek the endorsement and approval of your library board, sup- 
ported by appropriate signatures, and submit the completed application to 
the Library of Michigan for consideration. 



Step Five: Collect and 
Analyze Evaluation Data 



Take advantage of the period prior to the announcement of grant 
awards by developing and testing data collection instruments or investigat- 
ing data sources. When appropriate, new methods should be tested before 
being used to actually collect the data for evaluation of the project. This 
pilot test is particularly important for surveys but it is a critical pan the 
design of many methods. Even if you are using existing records, try out 
your procedures on a small portion of the records first to see if you can lo- 
cate and interpret the data that you want. Following the pilot test, review 
the experience with others in the stakeholder group before proceeding. 
Make any necessary changes in the instruments and procedures before 
moving to the stage of actual data collection. 

Whichever methods you use to collect data, respect the privacy and the 
rights of the individual. You may ask for permission when using data that 
could be considered confidential, and if you promise confidentiality then 
make sure that effective procedures are in place to guarantee it. 



Step Six: Summarize 
Evaluation Data for 
Stakeholders 




Following the completion of all data collection, select and organize the 
data according to the evaluation questions. Collate the data from each 
source and method, such as questionnaire surveys and individual inter- 
views, under each question. 

The procedures for analyzing data differ depending on whether the in- 
formation you collect is quantitative or qualitative. In general, quantitative 
data are used to measure the extent of something this is reported numeri- 
cally, for example the number or percentage of people who gave each an- 
swer on a questionnaire; or the number of new library cards issued, the in- 
crease in interlibrary loan requests processed, or the number of database 
searches conducted. 

Qualitative data is gathered through open-ended answers to interviews, 
questionnaires and narrative observations of events, and can be categorized 
to answer the evaluation questions. It is based on values, not numerical 
data. For example, seniors’ comments about a new collection might be cat- 
egorized as one of the following: “helpful with retirement,” “entertaining 
reading,” or “does not have what was wanted.” In doing data analysis, you 
muse take the quantitative and qualitative information you have collected 
and determine its significance in both numerical and narrative forms. The 
analysis should be presented in a way that answers the project’s evaluation 
questions. Consider reviewing the resources listed in the bibliography of 
this handbook for assistance in selecting the appropriate data analysis 
method. 

To aid your stakeholder group in interpreting the project data, you will 
need to summarize the information simply, accurately, and clearly. Identify 
key themes that are suggested by the findings. Providing charts and graphs 
to express data from lists and counts will also help stakeholders understand 
the information. 
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Step Seven: Prepare Final 
Evaluation Report 



The stakeholders should be involved in the interpretation of the find' 
ings and in reacting to preliminary and final reports. Find out what indi' 
vidual stakeholders believe are the implications of the data. Their involve^ 
ment will add credence to your overall findings and will create a sense of 
stakeholder ownership in the results of the evaluation and in the program. 
Stakeholder involvement at this point can also bring added visibility to 
the program when emphasized with a press release or other publicity tech' 
niques. The original evaluation questions developed by this group at the 
beginning of the project can serve as the guide for this discussion. The 
stakeholder group’s final report will then be included in the report sent to 
the Library of Michigan and to other appropriate audiences. 



Wrapping Up 



You may want to consider meeting with the stakeholders again to plan 
future actions for implementation of their recommendations. These might 
include continuing certain activities, changing others, or working to create 
even greater public awareness of the program or service. Continue to 
evaluate an ongoing project or program with help from the stakeholder 
group. 

If the stakeholder evaluation design process has worked well, the library 
staff, users, and people from the community will be interested in the find' 
ings. They will use the findings as evidence for continuing, improving, ex- 
panding, or discontinuing the project. The thorough evaluation you have 
done will help to assure that an appropriate response is made, and the 
reputation of the library for sound management will be enhanced. 
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Appendix A 



Library Project Planning Meeting 

Tentative Agenda 



1. Introduce Meeting Participants 

2. Review Purpose of Meeting 

• Identify Source of Grant Funds 

• Discuss Community Need in Relationship to Grant Funding Categories 

• Present Overview of Project Concept 

• Describe Stakeholder Evaluation Process 

• Summarize Efforts to Date 

3. Describe the Proposed Project 

• Goal and Objectives 

• Activities 

• Budget 

• Evaluation 

4. Discuss and Complete Project Evaluation Worksheet 

5. Determine On-Going Role for Meeting Participants 

6. Schedule Next Meeting Date 

7. Adjournment 
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Provide complete answers to the following questions. Answers should reflect the input of the stakeholder group 
meeting participants: 
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Data Collection Methods 


Appendix C 


Methods 


Description 


Advantages 


Disadvantages 


Existing Records 

(e.g., registration materi- 
als, financial records, us- 
age counts) 


The library staff may already be col- 
lecting information through the nor- 
mal procedures of the library. By 
modifying current procedures or sim- 
ply examining the data in a different 
way, the library staff may be able to 
answer some of the key evaluation 
questions. 


Provides quantifiable 
evidence of activities 
and results. 

Requires minimum time, 
effort, and money. 


Analysis can be compli- 
cated. 

Data can be misleading if 
originally collected for 
different purpose. 

Usually reflects quantity, 

! not quality. 


Observation 

(e.g., structured participa- 
tion, casual conversation, 
counting session atten- 
dance) 


The library staffer designated pro- 
gram participants may be able to look 
for specific data in a consistent way 
that will help to answer the evalua- 
tion questions. The observer, through 
informal interviews or other means, 
could focus on such aspects of the 
project as project setting, nature of in- 
teractions, program activities, library 
user behaviors, informal interactions, 
unplanned activities, and unexpected 
behavior. 


Requires minimum 
interruption to program 
activities. 

Provides check on reports 
of staff and users. 

Provides context for un- 
derstanding other data. 


Observer must be skilled 
in process obsepi'ation. 

Observer filters data 
through individual per- 
spective and values. 

Program participants may 
not be open and natural. 




Methods 


Description 


Advantages 


Disadvantages 


Individual Interviews 

(e.g., structured interviews 
with individual partici- 
pants, face-to-face or by 
phone) 


The library staff may develop struc- 
tured questions for interviews with 
key people. The interviews with indi- 
viduals should consist of a series of 
questions designed to provide the 
specific information needed. The in- 
terview can be conducted face-to- 
face or by telephone. Designing effec- 
tive questions and conducting a good 
interview is not an easy task and may 
require an experienced interviewer. 


Can probe for meaning of 
responses. 

May create participant 
willingness to disclose 
sensitive information. 

Can control when and 
how questions are asked. 


Time consuming. 

Analysis may be difficult. 

Requires skilled inter- 
viewer. 

Some participants may 
feel threatened. 


Group Interviews 

(e.g., focus groups, nomi- 
nal groups, work groups) 


The library staff may collect informa- 
tion from small groups of people. A 
focus group (8-1 2 persons) is used 
when group interaction is likely to in- 
crease the quality of the data being 
collected. If possible, the group 
should be led by a skilled group fa- 
cilitator and another person should 
document comments of group mem- 
bers, the consensus of the group on 
specific Issues, and any observations 
about group member interactions. 


Stimulates thinking and 
sharing ideas. 

Can get different views on 
same subject. 

Can get consensus about 
a program. 


Cannot be confidential. 

May be difficult to 
organize. 

Some participants may 
feel threatened. 

Requires skilled group in- 
terviewer. 



II 
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Data Collection Methods (Continued) 



Methods 


Description 


Advantages 


Disadvantages 


Journals/Logs 


The library staff could be asked to 


Provides record of imme- 


Data is subjective and 


keep a record of anecdotes, observa- 
tions, personal reactions, comments. 


diate reaction to events. 


not as reliable. 


(e.g., participants self-re- 


and the frequency of specific activi- 


Provides record of change 


Can be difficult to 


ports and critiques of ex- 
perience) 


ties relating to the project. The pur- 
pose of using this method is to un- 


over time. 


analyze. 


derstand the experience of the 


Requires minimum effort 


Participants must be 




program from the participant's point 
of view. The information collected, 
however, will be subjective and may 
be difficult to analyze. 


to collect data. 

Provides record of unan- 
ticipated events. 


trained in how to record 
information. 


Questionnaire Surveys 


The library staff can use question- 


Can collect data from 


Data is restricted by the 


naires in obtaining information from 
a large number of individuals. The 


large number of people in 
short time. 


questions that are asked. 


(e.g., systematic data col- 


nature of the information to be col- 


Relatively inexpensive. 


Substantial planning time 


lection instruments: paper 
and pencil, telephone, 


lected should be easily categorized. 
Surveys may be mailed or hand-de- 


is required. 


computer) 


livered, person-to-person, or tele- 
phone surveys. Questionnaires that 
collect accurate data are not simple 
to design and adnriinister. One may 
wish to use an experienced indi- 
vidual in designing the survey. Time 
should be taken to obtain feedback 
from stakeholders and a few people 
in the target population should test 
the survey before finalizing. 


Can be confidential. 

Provides ease of analysis 
and summarizing of data. 


Return rates can be 
quite low. 



Methods 


Description 


Advantages 


Disadvantages 


Expert Opinion 


The library staff may contact people 


Is relatively simple to de- 


Experts may not be avail- 


who are knowledgeable because of 
their experience and expertise in the 


sign and inexpensive to 
implement. 


able for this. 


(e.g., panel of library 


content or process of the project that 


Strengthens lines of com- 
munication between ex- 


May be difficult to find 


directors) 


is being evaluated. These people 
could be national experts, university 


neutral experts. 




faculty, or directors of successful 
programs in other libraries. Such per- 


perts and participants. 


Participants may not be 
directly ' imiliar with pro- 




sons can help assess the needs for 
new projects and programs; react to 
the quality of existing or planned 
programs; and suggest ways to im- 
prove programs. This expert opinion 
can be gathered through direct con- 
tacts with the individuals or through 
a r n/iew of reports and articles that 
they have written. 


Brings visibility to project. 


gram. 



O 




17 



Library of Michigan 
Stakeholder Evaluation Handbook — 1994 



Appendix D 



Bibliography 



Brinkerhoff, R.O., Dale M. Brethower, Terry Hluchyj, and jeri Ridings Nowakowski, Program Evalua' 
tion, Hingham, Massachusetts, Kluwer Nijhoff Publishing, 1983. 

Fink, A. & Kosecoff, J. How to Conduct Surveys: A Practical Quide for Applied Research. Newbury Park, Cali- 
fornia: Sage, 1988. 



McClure, C.R., Planning and Role Setting for Public Libraries. Chicago: American Library Association, 1987. 
Patton, M.Q. Creative Evaluation. Newbury Park, California: Sage, 1987. 

Patton, M.Q. How to Use Qualitative Methods in Evaluation. Newbury Park, California: Sage, 1987. 

Rosenberg, P. Cost Finding for Public Libraries. Chicago: American Library 
Association, 1985. 

Singleton, R., Straits, B.C., Straits, M.M., & McAllister, R.j. Approaches to Social Research. New York: Ox- 
ford University Press, 1988. 

Tice, K.E. & Gill, S.J. Education Information Centers: An Evaluation, journal of Career Development, 
Vol. 18(1), Fall 1991, pp. 37-50. 

Turock, B.j. Library Programs: Evaluating Federally Funded Public Library Programs. U.S. Government Print- 
ing Office, July 1990. 



Van House, N.A., Lynch, M.]., McClure, C.R., Zweizig, D.L., & Rodger, E.j. Output Measures for Public Li- 
braries. Chicago: American Library Association, 1987. 



Library of Michigan 
Stakeholder Evaluation Handbook — 1994 



Appendix E 



AdditUrrud Roles for Stakeholders 





In addition to the role that stakeholders will play in identifying 
evaluation questions, data resources and data collection methods, indi- 
vidual members of the stakeholder group or a smaller advisory commit- 
tee may serve in the following roles: 

♦ Function as a Planning Committee 

♦ Develop evaluation instruments 

♦ Pilot test evaluation instruments 

♦ Assist in evaluation implementation and data collection 

♦ Provide “reality checks” 

♦ Participate in project promotion 

♦ Assist with problem solving for the project 

♦ Provide interim evaluation analysis and advice 
on project revision 

♦ Pursue future project funding 

♦ Assess long-term effects or impact of the project 

♦ Identify needs and propose future projects for consideration 

♦ Provide a cadre of individuals which may be called upon to serve on 
the library boards friends group or ad hoc committees 
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Appendix F 



Qlossary 



Evaluation 

A method or methods of measuring the level of success of a project based on the collection and 
analysis of quantitative and qualitative information. Evaluations should answer the questions “so 
what?” or “what difference did the project make?” 

Funding Areas 

LSCA project categories which are eligible for LSCA funding. Specific funding areas are desig- 
nated by the Library of Michigan as being active for each fiscal year. 

Goal 

A general statement which describes the project’s broad overall intent. 

Methods 

Statements describing how project objectives will be accomplished. 

Objectives 

Specific statements identifying what will be achieved during the life of the project. Each objec- 
tive must be directly related to the project goal and will, ideally, be expressed in measurable terms. 

Stakeholder 

An individual who has a significant concern or “stake” in the success of the project being evalu- 
ated, e.g., teacher, city manager, library user. 





The Library of Michigan is committed to the effective evaluation of LSCA projects. This has 
been reflected by encouraging LSCA applicants to use the stakeholder evaluation method. The 
Library' of Michigan is participating in the national Evaluating Library Programs and Services 
project, coordinated by the University of Wisconsin^Madison. This project is developing the TelL 
It model of ev. luating library programs. TelLIt is similar to the stakeholder approach, however, it 
presents a broader framework for evaluation. Next year, the Library of Michigan’s evaluation 
handbook will be expanded to incorporate the TelMt approach. In future years, the focus on 
evaluation will continue to be a priority and new evaluation approaches will be included in the 
I-ibrary of Michigan’s LSCA program. 
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